【Task.31】feat(sft): add pair-aware DPO training - #257

# 🐛 Bug Fix

## Reject --ref-load under preference objectives

- Standard DPO snapshots the frozen reference from the initialized
  policy and rebuilds it from --hf-checkpoint on resume; --ref-load
  was silently ignored on that path while also rerouting the bridge
  mode policy-init fallback, so validate_preference_args now fails
  fast when it is set (RFC redai-studio#208: --ref-load is not the v1 DPO
  reference source)
- Drop the misleading --ref-load from the DPO recipe; behavior is
  unchanged because the bridge fallback already resolves to the same
  HF checkpoint

---

# ♻️ Refactor

## Deduplicate the preference-mode predicate

- Route data.py get_data_iterator and loss.py loss_function through
  is_preference_mode() instead of inline loss_type/sft_objective
  checks, keeping objective dispatch on a single source of truth

---

# ✅ Tests

## Cover --ref-load rejection

- Parametrize standard and reference-free DPO rejection cases in
  test_preference_runtime.py

# ♻️ Refactor

## Remove unreachable DPO loss fallback

- Drop the dead `0 * logits.sum()` branch in dpo_loss_function:
  build_preference_pair_indices already raises on an empty micro-batch

## Attach stable reason codes at preference raise sites

- Introduce _PreferenceRowError carrying an explicit reason_code from
  every normalization/split/truncation rejection; message matching in
  _classify_preference_error remains only as a fallback for errors
  raised outside the module, so reworded messages can no longer
  silently degrade rejections to "schema"

---

# ⚡ Performance

## Merge DP pair-count collectives

- Reduce _get_preference_data_iterator from three all_reduces to two
  by riding MIN/MAX on a single MAX all_reduce over [count, -count]

## Skip redundant same-tag model restores

- _switch_model now returns early when the target tag is already
  active, eliminating the duplicate full-weight CPU->GPU restore after
  the ref-forward finally block; paths that deliberately dirty weights
  already clear the tag first (covered by the injected-loader-failure
  reference integrity test)

---

# 📝 Documentation

## Note probe bitwise-determinism prerequisite

- en/zh DPO guides now state that the reference probe digest assumes
  an identical GPU/driver/image/kernel stack on resume, and that a
  probe mismatch signals environment drift rather than data corruption

## Justify the checkpoint-save barrier

- Comment that the post-sidecar barrier is functional (peers must not
  pass before rank 0 persists the identity file), and use an explicit
  gloo group for the rank-0 check per distributed code rules

# 🐛 Bug Fix

## Preserve standard DPO likelihood semantics

- Reject non-unit or non-finite rollout temperatures before actor construction
- Replace CUDA-to-Python condition checks with device-side asynchronous assertions
- Consolidate finite-value validation on the final DPO logits

## Validate preference data preparation inputs

- Validate message objects, roles, and string content on both preference branches
- Classify non-object source rows as schema rejections

---

# ✅ Tests

## Cover validation regressions

- Test finite and non-unit temperature failures
- Verify CUDA conditions avoid Python boolean conversion
- Cover malformed chosen and rejected message schemas

# 🐛 Bug Fix

## Pin standard DPO frozen reference provenance

- Resolve the reference from the declared repository and revision in the configured HF checkpoint directory
- Require Hugging Face local metadata to verify the pinned snapshot without downloading during actor startup
- Rebuild the frozen reference from that verified directory for both fresh starts and resumes
- Fail clearly when the configured checkpoint is unavailable, unverified, or resolves elsewhere

---

# 📝 Documentation

## Document pinned local model preparation

- Show the fixed-revision hf download command used by the public DPO recipe

---

# ✅ Tests

## Cover local reference resolution

- Verify repository, revision, local directory, and local-only cache resolution
- Verify missing metadata and mismatched resolved directories fail before model loading

# 🐛 Bug Fix

## Require pinned local snapshot metadata

- Require an immutable 40-character commit SHA for standard DPO references
- Verify Hugging Face local-dir tree metadata before accepting the configured checkpoint
- Reject copied or incomplete local snapshots that lack pinned provenance metadata

---

# ✅ Tests

## Cover metadata validation

- Exercise the pinned tree lookup and missing-metadata rejection paths
- Document the full commit-SHA requirement for the DPO recipe

# 🐛 Bug Fix

## Validate pinned Hugging Face local snapshots

- Parse the per-file metadata format used by pinned huggingface_hub 1.7.2
- Require every reference file to match the configured commit revision
- Verify regular Git files with Git blob SHA-1 and LFS files with SHA-256
- Stream large-file hashing to keep reference validation memory bounded

---

# ✅ Tests

## Cover reference provenance and integrity

- Exercise Git and LFS ETag validation
- Reject missing or mismatched local metadata
- Reject replaced weights even when their original mtime is restored

# 🐛 Bug Fix

## Enforce local reference completeness

- Require a supported single-file weight or standard Transformers weight index
- Validate non-empty index weight maps, safe shard paths, and format-specific suffixes
- Reject indexes that reference missing model shards

---

# ✅ Tests

## Cover missing reference weights

- Accept a complete multi-shard safetensors index
- Reject snapshots without a supported weight entry point
- Reject a deleted shard even when its local metadata is also removed

# 🐛 Bug Fix

## Move preference scheduling to the CPU control plane

- Replace CUDA count reductions and scalar readbacks with one DP-Gloo control gather
- Preserve pair-row, global-denominator, and micro-batch-count agreement across ranks
- Require Gloo process groups for preference objectives and enable them in the DPO recipe

---

# ⚡ Performance

## Remove hot-path GPU synchronization

- Eliminate all item calls and control tensors from the preference iterator
- Avoid GPU-to-CPU synchronization for train, reference, and evaluation iterators

---

# ✅ Tests

## Guard preference iterator contracts

- Cover unequal DP pair-row rejection through the Gloo control path
- Assert the iterator contains no item or all-reduce scalar readback path

# ♻️ Refactor

## Keep one reference identity path

- Remove the unused helper that reads the already-backed-up ref weights
- Preserve rebuild validation and backup ordering without duplicate digest work

---

# ✅ Tests

- Pass the full pre-commit suite
- Compile the Megatron actor module

# ✅ Tests

- Delete the fixed-source-hash TransferQueue sampler test as requested.
- Remove its now-unused imports without adding a CI-specific skip path.
- Fix the independent prefetch logging race by installing the capture hook
  before starting the worker thread.
- Focused CPU regression: 66 passed with the CI TransferQueue stub.
- pre-commit run --all-files --show-diff-on-failure passes.

Co-authored-by: Codex <noreply@openai.com>